1. Analysis of Spatial Image Rendering

Spatial image processing, such as Retinex, ACE, spatialfrequency, or bilateral, filtering, use the entire image in rendering scenes. These algorithms process captured scene radiances as input; then use the spatial information to synthesize a new image for rendition to a display or print. Spatial algorithms have different properties from pixel-processing algorithms. Pixel processes apply the same transform to all image pixels, so that all pixels with the same input value (i) have the same output value (o). However, spatial algorithms can convert identical input values into different output values. We discuss techniques most appropriate for measuring the success of spatial algorithms. We would like a simple figure-of-merit calculation for our favorite algorithm. We found that goal impractical. Spatial color algorithms are in the middle of the imaging chain, and their success is affected by preand post-processing. There are a variety of distinct goals for different spatial algorithms: one is to find the objects’ reflectance; one is to find the illumination; one is to make the best HDR picture; another is to model human vision. As well, there are different ground truth goals for each type of algorithm. Instead of presenting a universal solution to evaluate all types of algorithms, we describe a number of steps measuring scene characteristics that evaluate spatial processes. We describe examples of a number of control and test experiments that are useful in quantitative evaluation of portions of the imaging chain. This paper provides test images, measurements of scene characteristics, and examples of a set of flexible tools for quantitative evaluations of spatial color algorithms. Quantitative measurements of spatial algorithms evaluate the true performance of the central spatial process. This paper works in parallel with that provides appendices for detailed data. Introduction We use spatial processing for modeling human vision, and rendering high-dynamic-range (HDR) scenes. Spatial processes are needed to render images that cannot be processed using single pixel (Tone Scale) approaches. In human vision we see that color appearance does not correlate with the quanta catch of the receptors. Appearance does not correlate with pixel value. [1,2] In HDR photography we find that spatial rendering onto lower range display media avoids truncating scene information by converting HDR scenes into LDR images appropriate for the media. In both cases, we use spatial information because pixel processes cannot solve the problem. The common point of the spatial algorithms is that computations build up the output from the spatial information in the scene. They start with the quanta catch of the sensor pixel and apply spatial computations to make new image renditions, as does human vision. The spatial color family includes all the various Retinex implementations that can differ quite remarkably in the way they transform the image and apply spatial processing. They include alternative spatial algorithms (ACE or RACE). They include image domain ratio-products, frequency based spatial filters, some tone rendering algorithms and bilateral filters. They all are nonlinear spatial transforms of the receptor quanta catch. In general spatial algorithms are applied to the captured scene radiance so as to render the scene data with improved image quality. Spatial color algorithms have been studied by many authors. < http://web.me.com/mccanns/Spatial/Processing.html.> Types of Evaluation There are both subjective and objective measurements of images. Subjective measurements ask observers to select the preferred rendition of an image. Objective measurements compare the algorithms output pixel values in the processed image with ground truth. Ground truth is the goal of the algorithm. There are a great many spatial algorithms, and they have a wide variety of goals. The ground truth for each algorithm is defined by its author, so one ground truth does not apply to all algorithms. A second choice about evaluation techniques is whether to use a single metric for all attributes, or a series that measure individual image characteristics. Although efficient, observer preference experiments cannot discriminate between the many attributes that contribute to the judgement and measure their relative importance. Although more difficult, a series of objective measurements can help our understanding. Beauty Contest An effective subjective technique for evaluating scene rendering is to ask a number of observers to select the preferred image among different candidates. This is how photographic film response functions were determined. The weakness of this technique is that it provides little feedback on the underlying principles of why the algorithm works. We can identify the most preferred image, but learn little about why it is preferred. Moreover the judgement can be affected by the display technique and setup. Large displays in a dark room has different visual stimuli than a small print. The monitor, its color gamut, its profile, the display luminance, the ambient light on the screen, the viewing angle, and the image's visual angle all influence the appearance of the array of calculated digits. Departures from Ground Truth If reproductions actually replaced the light coming from the scene with an identical stimulus, then scene radiance would be the ground truth of image reproductions. Error metrics, such as the mean-squared-error comparing light from the scene, and that from the reproduction would be simple and effective. The problem is that reproduction media have response functions that transform the scene into a preferred rendition. Photographs do not reproduce scenes accurately. The preferred rendering is very different from scene radiances.[3] In order to perform a mean-squared-error calculation, model output vs. ground truth goal, we need information about the model’s goal. For human vision, ground truth is the appearance of the objects in the scene. For spatial algorithms that calculate objects’ reflectance, or for those that calculate the illumination on a scene, there are measurable ground truths, namely, the set of physical measurements of reflectance and illumination. The ground truth for the best-preferred reproduction of scenes has no universal definition. Rendition Quality Metric Can we find an objective analysis of image rendering using error metric analysis? First, we would need to measure the error for each pixel in a complex image. The error is a distance between the spatial processed output and the ideal ground truth. We also need to compare these errors in a uniform color space. In such a space, apparent changes in hue, lightness and chroma are all equal to numerical distances in the 3-D space. In uniform spaces, such as Munsell, the distance in the space represents the size of the change in appearance, while in XYZ, RGB, and sRGB spaces distance does not equal change in appearance. In addition, camera digits follow sRGB guidelines, but do not always follow the standard in regions near the limits of their color space along the color gamut. Built-in color enhancement firmware distorts these near-gamut regions of color space. In order to accurately convert camera digit to colorimetric XYZ, one needs detailed proprietary information of the signal processing in each camera. It is impractical to assume that we can transform the rendered camera response back into scene XYZ values and then convert them into an accurate, uniform color space coordinates. The second problem is that we need a goal image; we need an array of perfectly rendered pixels. How does one find the ideal rendered image of the scene? Algorithms that calculate physical quantities, such as reflectance and illumination, have well-defined ground truths. Algorithms that calculate appearance, or seek to make the best, most preferred reproduction has to find a quantitative description of appearance, or preference, to use objective analysis. We can use the beauty contest techniques for finding most preferred individual image, but in such experiments we find that the conclusions are image dependent. [4] Although it would be desirable to analyze images using error metrics, we see that the camera digit rendition of the scene is not equally spaced and that ground truth depends on the goal of the algorithm. Consequently, the single, universal objective measurement of combined properties is not practical. Nevertheless, we need objective evaluation techniques to measure the effectiveness of algorithms. Instead of combining all the properties of algorithms into a single metric value we can learn a great deal from the study of individual image-processing characteristics. For this we need scenes with known radiometric values. 3-D Mondrians This work is based on a series of experiments from the CREATE project.[5,6] The set of experiments used a scene with a Low-Dynamic-Range portion next to a High-Dynamic-Range portion in the same room at the same time. Both LDR and HDR parts were made of wooden blocks painted with 11 different paints. The LDR blocks were placed inside an illumination cube so as to be as uniform as possible. The HDR blocks had two highly directional lights. This 3-D test target has been measured with a wide variety of techniques: measurements of objects (reflectances); the light coming from 104 facets (XYZ); multiple exposure photographs using a number of different cameras; magnitude estimates of appearance of block facets; and watercolor paintings of the entire scene as a measure of appearance. We employed a number of these scene measurements to discuss possible evaluation techniques of spatial image processing. Spatial Color Examples Figure 1 is examples of images from the 3-D Mondrian experiments: normal digital images, spatial processed images, and Carinna Parraman’s watercolor paintings. Figure 1 shows LDR (top row) and HDR (bottom row) parts of the scene. The columns show normal digital photographs (left); the Vonikakis spatial image processing (center); and watercolor rendition of appearance (right). Comparison 1 (Figure 1) shows different renditions of LDR and HDR CREATE scenes (rows). The left column shows control photographs taken with a Panasonic DMC FZ5 digital camera (top, LDR; bottom HDR). The middle column shows the LDR and HDR outputs of a spatial algorithm (VV). The right column shows the Carinna Parraman watercolor painting of the scenes (rendition of scene appearance). Figure 2 show the normal digital photographs (left); the HP 945 Retinex spatial image processing (middle); and watercolor appearance (right). VV is a center-surround image-processing algorithm, which employs both local and global parameters.[7] The local parameters, which significantly affect the new value of a pixel, are its intensity (center) and the intensity of its surround. The global parameters that affect the overall appearance of the image are extracted from image statistics. The surround is calculated using a diffusion filter, similar to the biological filling-in mechanism, which blurs uniform areas, preserves strong intensity transitions and permits partial diffusion in weaker edges. New pixel values combining local and global parameters, are inspired by the shunting characteristics of the ganglion cells of the human visual system. The algorithm is applied only to the Y component. Comparison 2 (Figure 2) shows LDR and HDR control images captured by an HP945 digital camera, their spatial processed image along with the Watercolor painting of the scenes. (HP945). The retinex algorithm is a menu selectable part of the image processing firmware (Digital Flash) in the HP945 camera. It is a multi-resolution retinex process with ratio limits, described by Sobol.[8,9] Both the VV and HP945 Retinex algorithms belong to a subset of spatial algorithms called Spatial Color Synthesis Algorithm (SCSA). These algorithms attempt to mimic vision. Measurements of LDR & HDR 3-D Mondrians The scene has 2 identical sets of 3-D painted color blocks using only 11 paints on 100 facets. The LDR half is in nearly uniform illumination and the HDR half is in highly directional illumination. Calibration measurements of the CREATE 3-D Color Mondrians are available at Table 1 list the measurements. Camera images are multi-exposures (jpeg) of the LDR & HDR portions of the scene. Measurements of appearances are spectral reflectance measurements of paints in LDR and HDR watercolors. The artist recorded the appearances of the scene in non-uniform illumination on the watercolor painting made in uniform illumination. The reflectances of the watercolor are a measure of the scene appearance.[5] Table 1 lists the data available on the web from the CREATE experiment. Scene Characteristic Analysis Instead of looking for a single, universal metric value, we need to break the problem down into a number of practical questions that are possible to implement, with objective measurements. We can use the above data sets to perform a number of different analyses that help us to understand the many characteristics of spatial image processing. Table 2 lists seven different comparisons that characterize the properties of the rendered image. Table 2 lists six characteristic tests of the image processing chain. Data Available Format Source Paint reflectance spectra, XYZ Spectrolino LDR radiances LDR XYZ Konica Minolta CS100 HDR radiances HDR XYZ Konica Minolta CS100 LDR camera digits (sRGB) Multiple exposures HDR camera digits s(sRGB) Multiple exposures LDR appearances spectra, XYZ Spectrolino HDR appearances spectra, XYZ Spectrolino Test Segment Data A Data B